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Abstract 

A Monte Carlo study compared the usefulness of six variable weighting methods for cluster 
analysis. Datasets were 100 bivariate observations from two subgroups, generated according to a finite 
normal mixture model. Subgroup size, within-group correlation, within-group variance, and distance 
between subgroup centroids were manipulated. 

Of the clustering methods examined, tiie flexible average clustering algorithm with 
p = -.15 or -.20 gave flie best recover^'. Of the remaining methods. Ward's method yielded tiie best 
recovery, followed closely by beta-flexible linkage (P = -.50) and SAS's EML algorithm. 

In the absence of variable weights, negative within-group correlation resulted in much poorer 
recovery for all clustering algorithms. The ACE weighting method of Art, Gnanadesikan, and Kettenring 
provided a net improvement in 17-24% of the datasets when used with better clustering algorithms. When 
used wifli tiie same clustering alogrithms, De Soete's ultrametric weighting yielded improved recovery 16- 
22% of the time. However, although ultrametric weigliting was more sensitive than ACE to negative 
within-subgroup correlation. Clustering based on principal components was less effective. Therefore, the 
ACE method is preferred overall. 

There is still room for improvement, however. Clustering with Mahalanobis distance based on the 
pooled within-group covariance matrix indicated that knowing the coirect covariance matrix would yield 
improved recovery (over ACE) approximately 10% of the time. 
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Often in educational and psychological research, a researcher is confronted by a population which, 
for lack of a better term, seems too heterogeneous. In such circumstances, the investigator is likely to 
hypothesize diat the population under study is actually composed of two or more relatively homogeneous 
subgroups. For example, several investigators have suggested diat the lunbrella term "learning 
disabilities" actually encompasses a variety of disorders , with different etiologies and strategies for 
treatment. 

In such circumstances, one may appeal to clinical insight to describe possible subgroups. 

However, to bolster such insight, the investigator may also turn to the statistical method of cluster analysis 
to try to isolate relatively homogeneous subgroups within the more heterogeneous population. Indeed, 
cluster analysis often has been used to try to identify putative subtypes of learning disability. 

Unfortunately, the applied investigator is faced with a myriad of clustering techniques, most of 
which present options and suboptions in the analysis. To guide these choices, numerous studies have 
examined various aspects of the clustering process, including comparison of clustering algorithms (e.g., 
Belbin, Faith, & Milligan, 1992; Blasrfield, 1976; Donoghue, 1994b, 1995; Milligan, 1979, 1989a; 
Scheibler & Schneider, 1985), the effect of various types of "error" in the data to be clustered (Milligan, 
1980), variable standardization (Milligan & Cooper, 1988; Barton, 1993), selection/weighting of irrelevant 
variables (De Soete, 1986, 1988; Milligan, 1989b; Donohue, 1994a), procedures to determine the 
number of clusters (e.g., Milligan & Cooper, 1985), and procedures to compare clustering solutions 
(Milligan & Cooper, 1986). 

Although these studies have obtained a variety of useful findings, one aspect of the clustering 
process has not received much attention: the within-group covariance stnicture. Recent work (Donoghue, 
1994b) examined the oise of two groups in two dimensions, and found a very large effect of within-group 
con-elation. Specifically, within-group correlation which did not coincide with the direction of separation 
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in subgroup means was associated with lower recovery for all clustering methods; within-group correlation 
which did coincide with the direction of separation was associated with higher recovery. This result was 
interpreted in terms of the similarity* measure used, die Euclidean distance. 

A fundamental issue raised by Donoghue (1994b) is what the analyst should do to minimize the 
deleterious effect of within-group correlation. Donohue suggested several possible alternatives to the 
Euclidean distance, but, in the absence of comparative data, could provide no guidance. The present study 
compares several of these variable wei^ting methods. In the remainder of this section, the finite mixture 
model which implicitly imderlies this work will be introduced, and the various weighting strategies used in 
this study will be examined. 

This paper adopts the view of cluster analysis as the attempt to "unmix a mixture of distributions" 
(e.g., Titterington, Smith, & Makov, 1985; McLachlan & Basford, 1988). Thi finite mixture model 
states tiiat the distribution function for the entire population, h(x), is given by: 

h{x) = X) ’ 

( 1 ) 

> 0. E = 1 • 

g-'- 

Subgroups are the homogeneous distributions f,(x), which are mixed, and the are termed the "mixing 
proportions." The conceptual framework of the finite mixture model of clustering underlies the present 
paper, and some of the results may not make sense in situations in which the finite mixture model does not 
apply. 



‘ The geiicral tenu ’^mcasuire of smiilanty" will be ilscxI in tliis paper to denote both true measures of sirnilRrily, 
such as correlations and coefficients of concordance, and measures of dissimilarity, such as distances. 




^ ; 

/ 
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In flie discussion tiiat follows, X will be die full (entities by variables) data vector, and Xj be the 
column vector of variable values associated with entity i (i.e, the transpose of the ith row of matrix X. 
Variahift ^Righting Methods 

Prior to clustering, hierarchical clustering mediods convert two-mode (variables by entities) 
multivariate data into a one-mode (entities by entities) tmivariate measure of similarity, which is dien used 
as die basis of the clustering. The vast majority of clustering studies (applied and simulation) me dg, 
Euclidean distance (or the squared distance) between entities i and j as similarity measure: 








( 2 ) 



where P is the number of variables used in die clustering. Basic geometry of vector spaces reveals that 
Euclidean distance is correct only if computed on an orthogonal basis. If die variables are correlated, the 
Euclidean distance can misrepresent the distance between two points. By ignoring the within-group 
correlations of variables, Donoghue (1994b) demonstrated that dg can seriously degrade cluster recovery. 

Euclidean distance is widely used and is the default measure for most clustering software. 
However, most hierarchical clustering algorithms can make use of any one-mode measure, and of ^;r 
similarity measures are available. Two commonly mentioned possibilities are Mahalanobis distance 
and clustering based upon principal component scores. The problems widi these obvious choices will be 
discussed, and then some possible alternatives will be discussed. 

Ideally, one would cluster based on the Mahalanobis distance measiue, 

~ - Xj) 



O 

ERIC 



(3) 
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in practice however, is difficult to compute because of the difficulty in estimating the appropriate S 
matrix. The correct choice is the pooled within-group covariance matrix, 

( 4 ) 

Without knowing fee subgroup structure, it is not possible to compute the 2^ matrices. An alternative is to 
xise the overall covariance matrix, Sj-: 

IV = . (5) 

n 

However, die standard MANOVA decomposition reveals: 

Sj. = . (6) 

The inclusion of Sg can give Sj. very different properties from Sw- Hartigan (1975, p. 63) provides an 
example demonstrating that D^, based on Up can result in worse cluster recovery than using dg. 

Anodier commonly suggested alternative is to cluster based upon principal component scores 
rather dian the original variables. The chief advantage of this method is tliat principal component scores 
are orthogonal to one another; because dg imolicitly assumes orthogonal variables, clustering based on the 
principal components seems to remove the objections to this measure. However, the difficulty with 
principal components is identical to that of Dm- The ideal is to perform principal components analysis 
based on S^y. In practice, however, is not knov/n, and so component scores are computed based on IV 
(or die correlation matrix R,-). As is the case wth D„, the inclusion of Sg can give very different 
properties from Sw Rohlf (1970) discusses some of the difficulties with using principal components, and 
Chang (1983) gives a theoretical analysis of the difficulties involved, and provides a clear example of the 
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problems wifli this approach. 

Thft ACF. Algnrithm 

Art, Gnanadesikan, and Kettenring (1982) devised an ingenious method to estimate the pooled 
within-group covariance matrix without knowledge of the subgroup structure. They start with the known 
property fliat die covariance can be computed from inter-entity relations: 

Analogous to the usual MANOVA decomposition in Equation (7), the inter-entity relations give a between- 
and within-groups decomposition: 

T = B' * W* . (8) 

If A is the matrix of differences between subgroup centroids, Art Gnanadesikan, and Kettenring show that 
W*'*A has the same eigenvectors as W’A, and so based on W* is equivalent (up to a multiplicative 
constant) to Dm based on W. 

Reasoning that most of the smaller inter-entity distancf;s are likely to be within-group distances. 
Art Gnanadesikan, and Kettenring use the inter-entity relations from the m smallest distances to form an 
approximate covariance estimate (ACE), W* : 

W"’ = - E (9) 

where die summation runs over the m smallest pairs. The distances are then recomputed as Dm based on 
W*, and the process ^ epeated imtil die current estimate of W* does not differ significantly from that of the 
previous iteration. 

A modified version of ACE is computed by SAS's PROC ACECLUS (SAS Institute, 1988), 
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including the option to output a matrix of inter-entity similarities for further analysis. The present study 
used the original algorithm given in Art, Gnanadesikan, and Kettenring (1982). To date, I am not aware 
of any published studies (beyond tiie original article) of this approach, although work is underway at 
Bellcore (J. Kettenring, personal communication, July 2, 1993). 



De Soete (1986, 1988) developed an algorithm which determines weights to apply to variables in 
computing a distance measure; 



= 



r 

E M'pCv ■ ’ 



^ p**i 



E = 1 • 

P^\ 



( 10 ) 



The weights are chosen so that the resultant distances maximally satisiy the "ultrametric inequality," which 
states that any three points i, j, and k, the distances between the points should satisfy tiie relation; 

d^. i maxid^, . ( 11 ) 



This is equivalent to requiring all sets of three points to lie on an acute isosceles (or equilateral) triangle. 
Johnson (1967) and Milligan (1979) demonstrated the relationship between the ultrametric inequality and 
many commonly usee hierarchical clustering algorithms, and Milligan and Isaac (1980) give simulation 
results which provide support for the utility of the conceptualization. Two studies (Milligan, 1989b, 
Donoghue, 1994a) have exan.!ned De Sck-^'s algorithm, and found that it greatly improved cluster 
recovei-y when the data contained "error" dimensions, i.e, dimensions which contained no information 
about subgroup mernbersltip. However, it is not known whether this method helps with the difficulties of 
within-gronp covariance structure. 
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The present study sought to examine these weighting strategies, and compare their usefulness in 
recovering known subgroup stn’.cture in data with the presence of a variety of widiin-group covariance 
structures. 



Method 

This study used Monte Carlo methods to systematically investigate the utility of various variable 
weighting methods to account for the effects within-group covariance structures. Subgroups were 
generated, according to a model of a finite mixture of normal dist 'butions (e.g., McLachlan & Basford, 
1988). Similar to Donoghue (1994b), the study will examine the limited case of two groups in two 
dimensions, but diis case will be examined in some detail. While restricting the investigation to bivariate 
data somewhat limits its generalizability, this provides a minimal test of the weighting procedures; any 
procedure which does not function well with bivariate data seems unlikely to be useful for higher 
dimensional cases. 

Design 

Five aspects of the data were manipulated in data generation. The first three of tliese determined 
die within-group covariance matrices. 

1) Rl: The within-group correlation of subgroup 1 (3 levels)- ti - -.7, 0.0, .7. 

2) R2: The within-group correlation of subgroup 2 (3 levels)— X 2 = -.7, 0.0, .7. 

3) COV: The relationship of the within-group covariance matrices (5 levels). For tlie first three 

levels, the variances of e.ach variable were equal (o^g = 0 ^ 2 ^ within a subgroup, and both of the 
vaiiances of the second subgroup were equal to a constant times the variance within the first 
subgroup (o",^ - o‘/r; = conditions, tlie vanauces of the two 
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variables within a subgroup differed. The witliin-subgroup matrices vere equal for one of the 



>^noitions, but differed for the other condition. The five levels of COV were: 




B)l:9-o^, = l.o\.= l,o^, = 9, = 9 




D) HO - 0^1 = 1, o\ 9, 0^2 = 1, 0^22 = 9 



E) HI - o^, = 1, - 9, 0^2 = 9, 0^22 = 1. 



In generating the data, two other "nuisance variables" were manipulated, due to their consistently 
large effects in other studies: 

4) PROS: Probability of each of the subgroups in the population (pi and P 2 ). This was manipulated 
by varying the sizes (ni = N^^pi, nj = N’^'-pj) of the subgroups (2 levels)— Equal sized groups 
(ni=50, n2=50) or unequal gioup sizes (ni=90, n 2 =10). 

5) DIST: Separation of subgroups. This was defined in terms of D^, the Mahalanobis distance (in 
tile populations) between the subgroup centroids: 



This distance was based on the pooled witliin-gioup covariance matrix: S = pi*E, + P2'*'22 DIST 
had 3 levels-Dw = 2, Dm = 4, or Dm = 6. 

6) WEIGHT: The method of weighting the variables to form the similaritj' measure (6 levels)— which 
are discussed in a separate section below. 

7) METHOD: The method of cluster analysis (1 1 levels)— \vhich arc discussed in a separate section 
below. 




( 12 ) 



The first five factors were fully crossed to yield 270 conditions. Twenty datasets were generated 
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for each condition according to the procedure given below, giving a total of 5400 base datasets. Although 
more replications would be desirable, results of previous work (e.g., Donoghue, 1994b) indicate that 
this number of replications is likely to provide sufficient precision for the purposes of this study. Each 
dataset was then weighted according to each of the weighting methods, and analyzed by each of the 
hierarchical clustering methods, yielding a total of 356,400 cluster analyses. For each cluster analysis, the 
solution for the correct number of subgroups (i.e., G=2) was used as the result for that clustering method. 

Data generation, variable weighting, and clustering were performed using FORTRAN programs 
written by the author. Eigenvalues and eigenvectors were computed using routines from EISPACK 
(Smith, Boyle, Garbow, Ikebe, Klema, & Moler, 1974). Accuracy of these programs was ensured 
through numerous comparisons of results of subroutines and final classifications with routines from SAS 
and SPLUS. 

DataJCensration 

Subgroups were generated according to a finite mixture of normal distributions. All datasets 
consisted of 1(X) observations. The means were separated by both variables equally. To ensure that the 
data generation procedures worked properly, the Mahalanobis distance between the two known groups was 
computed for each dataset. Table 1 presents descriptive statistics for Dm. 

Insert Table 1 about here 

The means and medians for each of the conditions are very close to the desired values, supporting the 
validity of the data generation procedures. 

Variable. Weight ing Methods 

For analysis, each dataset was subjected to 6 variable weighting procedures. The procedures 
either yield a set of weighte to apply to the variables before forming the Euclidean distance, or they yield 




i 



t 
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an alternative similarity measure such as Dm- 

1) Euclidean distance (no weighting) 

2) PC-R: Principal component scores based on Rt 

3) Dm-T: Dm based on Sj 

4) Ultrametric-dr computed using weights from De Soete's (1986, 1988) algorithm to maximize 
agreement with the ultrametric inequality 

5) ACE: Dm based on W* from the ACE algorithm (Art, Gnanadesikan, & Kettenring, 1982). As in 
Art et al., the parameter m (the number of inter-entity distances used to detemune the pairs used 
to estimate W*) was set at two-thirds of the number of within-group pairs: m = 1633 for pi = 0.5 
and m — 2700 for pi = 0.9. 

6) Dm-W: Dm based on pooled within Sw 

A seventh method, principal component scores based on St, was originally included. However, because 
both component scores were retained, these scores are an orthogonal rotation of the original variables. 
Because the Euclidean distance is invariant under orthogonal rotation, every one of the 59,400 solutions 
was identical to that computed from d^, and so these results were not included in the analyses discussed 
below. Clearly, clustering based on principal components is not the solution to the problem of within- 
group coiTelation. 

The first and last methods provide useful comparisons for fee other methods. Clearly, a weigliting 
method which results in worse cluster recovery is undesirable. On fee other hand, the last method 
provides a useful upper bound on how good recovery could be in a given dataset. 

OusteLAlgorilhms 

Each dataset was analyzed 1 1 times, corresponding to different hierarchical clustering algorithms. 



The clustering methods are: 
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1) Single linkage, distance 

2) Complete linkage, distance 

3) Ward's (1963) method, squared distance 

4) EML-SAS's (1988) maximum likelihood hierarchical clustering procedure, which is a 

modification of Ward's method to alleviate the method's tendency to yield equal-sized 
clusters, squared distance. 

5) Average flexible method, (Belbin, Faith, & Milligan, 1992), distance. Five levels of p were 

examined, 0.0, -.10, -.15, -.20, -.25. Note that P = 0 is the usual average linkage 
method. 

6) Flexible-beta method (Lance & Williams, 1967), distance. Two levels of P were examined, 

-.25 and -.50. 

The clustering methods were chosen because; a) they are widely used (single linkage, complete 
linkage, average linkage, and Ward's method); b) they have performed well in previous studies (average 
linkage. Ward's method, beta-flexible and flexible average); or c) they are designed to rectify a known 
w^kness of another algorithm (EML). The values of P used are based on the studies by Milligan (1989a), 
Belbin, faith, and Milligan (1992), and Donoghue, 1994b, 1995). For a discussion of these algorithms, 
the reader is referred to standard introductions to cluster analysis (e.g., Everitt, 1993; Lorr, 1983). 
Milligan (1989a), Belbin, Faith, and Milligan (1992), and Donoghue (1995) contain discussions of tlie 
beta-flexible and flexible average methods, and the SAS documentation (SAS Institute, 1988) is the 
primary reference for the EML algorithm. 

.Outcome-Measure 

The outcome measure for the study w'as the Hubert and Arabic (1985) modification of Rand's 
(1971) statistic, which rvill be denoted HA-Rand. The index was coinputefl Ixjfiveen fiach cluster solution 
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and the true subgroup membership used to generate tlie data. This index is based on examining pairs of 
entities, and determining whether they are classified into the same or different subgroups. A value of zero 
reflects chance agreement with the true membership, and 1.0 reflects perfect agreement. A study by 
Milligan and Cooper (1986) supports the accuracy of Hubert and Arabic's modification. 

Analyse s 

To summarize the results, a factorial analysis of variance was conducted, as in Milligan (1980, 
1981, 1989a) and Donoghue (1994b, 1995). The HA-Rand index served as the dependent variable. The 
independent variables were the design factors used to generate the data, weighting metiiod, and the cluster 
algorithm used to analyze the data. The purpose of the ANOVA was to sununarize the data and help to 
highlight die more important effects. Therefore, a measure of effect size was adopted in favor of 
traditional significance testing. Usually, would be used in this context; 



However, this index has a disadvantage in large designs, namely that the denominator contains not only 
eiTor variance and systematic variance of interest, but also irrelevant systematic variance due to other 
effects. The larger the design becomes, the more apparent this effect becomes. In the present case, this is 
particularly noisome, because one of the factors, Malialanobis distance between the subgroup means, has a 
very large effect and so serves to obscure the effects of other factors. Therefore, an alternate version, 
(Tabachnick & Fidell, 1983, p. 47), was used; 




SS 






SS 



+ SS 



( 14 ) 



C^CCt 



Note that, vmlike tire other formulation, this version, ri\j„ does not sum to 1.0. As in Donoghue (1994b), 
the practical criterion of .03 was selected. Note that, given the large amount of data, any effect 
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which met the practical criterion was always highly significant (e.g., p < .0001). 

Results 

The results of the ANOVA and values of are summarized in Table 2. All of the main effects 
were identified as salient, with the exception of Prob. and R2, the correlation within subgroup 2. Means 
for the main effects are presented in Table 3. The main effects are all modified by salient interactions, and 
so will not be discussed in detail. However, a few comments are in order, bearing in mind that the 
observations may well be modified by the interactions. First, die extremely large effect of D, the distance 
between the subgroup centroids, is both noteworthy and expected; as the subgroups become more 
separated, the clustering task becomes easier and hence recovery improves. The smaller effect for R2 
(compared to Rl) is also an expected artifact of the design; for one-half of tlie datasets, subgroup 2 
comprised only 10% of the sample, and so had a smaller effect. Finally, for the weighting methods, Table 
3 reveals fliat, overall, best recovery is obtained for Dm-W, the Mahalanobis distance based on the pooled 
within-group covariance matrix. This is followed by distances based on the approximate covariance 
estimate, ACE; distances computed from the ultrametric weiglits; and PC-R, the principal components 
based on the total correlation matrix. Overall, D^-T, Mahalanobis distance based on the total covariance 
matrix, yielded worse recovery than simply using the Euclidean distance de. 

Ip' ' ' ■ i ::,b!: s 2 and 3 about here 

Walghtixig-Method 

The main effect of weighting method was modified by several salient interactions. Figure 1 
poviviiys tiic Rl by Weight interaction. Tliree of the weighting methods, dg, PC-R, and the ultrametiic 
weights show large drops in recovery when Rl •= -.7, The other three methods based on 
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ACE, and Dm-T) show a relatively flat profile; recovery using ACE actually decreases slightly as R1 
increases. Although ultrametric weights and PC-R sliow lower recovery for R1 =-.7, both methods yield 
good recovery for R1 =0 and R1 = .7, outperforming ACE. 

Insert Figure 1 about here 

Figure 2 plots the means of COV by Weight interaction. The effect appears to be due to the 
mediods which are not based on Mahalanobis distance methods, dg, PC-R and the ultrametric weights. 

The chief effect seems to be for HO, the condition in which On= 02 i= 1 and 0i2=02>=3. The ultrametric 
weights show a small decrease in recovery for this condition; PC-R has a substantial decrement in 
recovery, and dg demonstrates a huge decrement in recovery. The three methods also differ to a lesser 
extent on HI, but give virtually identical results for the other conditions. 

Insert Figure 2 about here 

The Prob. by Weight interaction is portrayed in Figure 3. Dm-W is relatively unaffected by 
subgroup size. PC-R r.nd ultrametric weights yield moderately lower recovery when pi — .9, and d^ 
yields substantially worse recovery. On the other hand, D^-T and ACE yield better recovery for pi = .9 
than for Pi = .5. For ACE, the number of pairs used in the estimation, m, was larger for pi = .9, so this 
result may simply reflect more stable estimation from more data. 

Insert Figui'e 3 about here 

Finally, Figure 4 displays the means for the Weight by Method interaction. Although there are 
some differences, five of the weighting methods give very similar profiles of recovery across the clustering 
algorithms. On the other hand, Df,,-T differs v.dldly in effectiveness. It performs well for Ward's method 
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and flexible average, but performs at or below the level of dg when used with other methods. Because 
these two factors (clustering algorithm and weighting method) are under control of the analyst, this 
interaction is of considerable interest. Tuerefore, it will be examined in more detail below. 



Three additional two-way interactions involving clustering method were identified as salient: 
Distance by Method, Prob. by Method, and COV by Method. Figure 5 portrays the means of the Distance 
by Method interaction. In general, the ordering of the methods is similar for each distance, although 
complete linkage ranks ahead of average linkage for Dm =2, but behind average linkage when the groups 
are further apart. The largest difference is due to the single linkage method, which shows a very small 
g ain (relative to other methods) from Dm = 2 to D„ 4, and a consequent large relative gain from Dm = 
4 to Dm = 6. Otherwise, the effects of increasing subgroup separation are relatively stable across the 
other methods. 



Figure 6 plots the means for the Prob. by Method interaction. The clustering metliods differ 
substantially in their recovery of equal- versus unequal-sized groups. Consistent with other studies (e.g., 
Donoghue, 1994b), single linkage yields much better recovery for unequal-sized groups, as do, to a lesser 
extent, EMI^ and complete linkage. On the other hand. Ward's method yields much better recovery for 
equal -sized groups. EML and Ward’s metliod showed a similar pattern in Donoghue (1994b), but there 
EML yielded slightly better recovery for equal-sized groups. 



Insert Figure 4 about here 




Insert Figure 5 about here 
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Insert Figure 6 about here 

Finally, the COV by Method interaction is portrayed in Figure 7. The 1:1, HO and Hi conditions 
present similar profiles of recovery across the clustering algorithms. The 1 :9 condition results in much 
hi^er recovery for flexible average, beta-flexible and Ward's methods but lower recovery for average 
linkage. In general, the 9:1 condition yields a similar profile to the 1:9 condition, although the 1:9 
condition results in worse recoverjf for single and average linkage. In spite of these differences, however, 
the clustering methods are, in general, similarly ordered for each condition of COV. 

Insert Figure 7 about here 



The two-way interaction of Distance by COV was identified as salient, as was the three-way 
interaction of Distance by Prob. by COV. Figures 8a and 8b plot the means of this three-way interaction. 
When Pi = .5 as in Figure 8a, the Distance by COV interaction is fairly small. For each level of D^, 
recovery is slightly lower for the HO condition of COV than for the other conditions, although the amount 
does differ somewhat by Dm. However, when pi = .9 as in Figure 8b, there is a substantial interaction of 
Distance with COV. For Dm = 2, the 1:9 condition results in good recovery, wliile the recovery for the 
9: 1 condition falls near chance, and HO and HI yielded recovery equal to or better than that of the 1:1 
condition. For Dm = 4, tlie 1:9 condition yields similar recovery to that of 1: 1 and 9: 1 , and for - 6 , 
1:9 results in somewhat lower recovery than the other two conditions. 



Insert Figure 8 about here 
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Other Weight hy Metbod Tnteraetinns 

Ins^ction of Figure 4 indicated tiiat the bulk of the Weight by Method interaction was due to the 
differing utility of D^-T, the Mahalanobis distance between entities computed from Sr- Table 2 indicated 
tha t two tiiree-way interactions involving Weight and Method (Distance by Weight by Method and Prob. 
by Wei^t by Method) were large but did not meet the criterion for salience. For completeness, plots of 
the means for these interactions are presented in the appendix. These interactions are largely due to the 
behavior of Dm~T; the interaction of with Method increases with subgroup separation, and the 
interaction is much larger for pi = .5 than for Pi = .9. 

What Works 

As was noted above, the Weight by Method interaction is of particular interest. While the other 
factors in the design are aspects of die data which are outside of the analyst's control, the weighting 
method and clustering algorithm are decisions made by the analyst. The key questions are: 

1) Which weighting method yields the best recovery? 

2) Which clustering algoritlim yields the best recovery? 

The salient interaction between Wei^t and Method indicates that the answer to either of these questions 
may depend on the value of die other factor. Essentially, we would like to know if the interaction is 
ordinal (effect size changes but the order of the levels of one factor remains stable across levels of the 
other) or disordinal (the ordering of results for one factor depends upon the level of the other factor). 

Because these are explicidy ordinal questions, Cliffs (1993) distribution free method of comparing 
two distributions was used. Ordinal comparisons were performed using a modified version of Cliff s 
(1992) program PAIRDELl, for paired observations. Two types of ordinal comparisons were made. The 
first estimates the probability that a randomly sampled observation from one distribution is higher (i.e., 
better recovery) than a randomly sampled observation from the other distribution. This residts in one of 




2 ^ 
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ftree decisions for each pair of metiiods; a) Method A is higher (better recovery) than Method B; b) 
Mefeod B is higher than Method A; or c) the methods do not significantly differ. These pairwise 
comparisons were made on HA-Rand index recovery values. Shaffer's (1986) modification to the 
Bonferroni procedure was used to maintain familywise Type I error at a =0.05. Finally, the pairwise 
relations were converted into ranks based upon die number of methods which were significantly higher 
dian a given metiiod versus the number which were significantly below it. 

The second type of ordinal comparison is based on Cliff’s index d^, which is the proportion of 
datasets for which Method A yielded higher recovery than Method B minus the proportion of datasets for 
which Method B yielded higher recovery than Method A; is the net proportion of datasets for which 
Method A yielded higher recovery. In the present context, it may be interpreted as the probability (for a 
randomly chosen dataset) of getting better recovery using Method A. Negative values of dy^ indicate lower 
recovery for Method A. 

To determine which weighting methods work better, pairwise comparisons of die weighting 
methods were computed within each clustering algorithm, and ranlcs were formed based on the pairwise 
comparisons. For each clustering algorithm, the initial family size was 6 take 2 = 15. The results of 
these comparisons are summarized in Table 4. 



Insert Table 4 about here 



In spite of the strong interaction, tlie ranks of weighting methods are very consistent across 
clustering algorithms; only du and D^-T shov/ reversals across metliods. Clustering using D^-W 
(Mahalanobis distance based on die pooled within-groups covariance matrix) yielded best cluster recovery. 
Of the remaining mediods, wliich do not require knowledge of the cluster stnrctuve, the ACE algorithm 
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{Art et al., 1982) is best, followed by the ultrametric weighting algorithm of De Soete (1986, 1988). 

As was mentioned in die introduction, the (squar»i) Euclidean distance dg is the default measure of 
similarity for most implementations of clustering procedures (e.g., statistical packages). Table 5 provides 
values for each of the wei^ting methods compared to dg. A negative value indicates that the method 
indexed at the top of the column gave superior recovery to that of dg. For example, using dg instead of 
ACE would yield lower recovery 8-24% (depending upon the clustering algorithm used) of the time when 
analyzing datasets similar to those used here. Table 5 also gives values for each of the remaining 
methods compared to the best method, ACE. In comparing the two best methods which do not require 
knowledge of the subgroup structure, ACE and ultrametric, the value varies somewhat by clustering 
algorithm, but is always greater than 0; for the type of data considered in this study, ACE provides better 
recovery than ultrametric weights. 

Insert Table 5 about here 



The last two columns of Table 5 compare cluster recovery using the two best weigltting methods 
(ACE and ultrametric weights) to cluster recovery using D^-W. Thus, knowing the subgroup structure 
and clustering based on Dv,-W instead of ACE would result in improved recovery 10-18% of tlie time, 
unless the analyst used tlie single linkage algorithm. 



A similar procedure was used to rank the clustering algorithms. Within each weight method, ali 
pairwise ordiital comparisons of clustering methods were performed and the pairwise relations were 
converted into ranks. Shaffer's modification to the Bonferroni was again used, with the initial family size 
of 1 1 take 2 - 55. The results of tirese anal’ ses are summarized in Table 6. 
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Insert Table 6 about here 

The ranks of the clustering methods are more variable than those of the weights. However, the 
flexible average method of Belbin, Faith, and Milligan (1992) clearly outperforms the other algorithms. 

For the conditions examined in this study, P = -. 15 or p = -.20 gave the best recovery. These values 
coincide witii Belbin et al.'s recommendation that P ^ -.2, althou^ they are slightly lower than the final 
value tiiey suggest, P — 10. However, for the best weighting conditions, P = -.10 yielded very similar - 

results to the best values. 

Althou^ there are some minor differences, the results for the remaining methods are similar to 
those obtained in other studies (e.g., Donoghue, 1994a,b; Milligan 1989a,b; Belbin, Faith, & Milligan, 
1992). The results for the lowest of the methods, complete linkage and single linkage, are quite consistent. 
Based on tiieoretical considerations, such as its relationship to the minimum spanning tree of graph theory, 
some authors continue to recommend single linkage (e.g., IMS Panel on Discrintunation, Classification, 
and Clustering, 1989). Yet this method uniformly produced the lowest recovery of the methods examined, 
a result that is routinely found in comparisons clustering algoritluns (e.g., Milligan & Cooper, 1987, and 
tlie references therein). 

Table 7 presents values for comparing each clustering method to one of the best (although not 
widely available) clustering methods, flexible average with P = -.15, and comparing each clustering 
method to one of the best of the widely available algorithms. Ward's method. In the present study, using 
the flexible average method instead of single linkage gives — .441 for di;, .590 for ACE, and .524 for 
ultrametric weights. Using s..«gle linkage gives worse recovery about one-Jialf of the time! Even 
compared to Ward’s method, which is very widely available, d„ is over .30 for all v,'eighting methods. 
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Insert Table 7 about here 



T.imitatinns anri Future Wnrlg 

The following is a very brief discussion of some related areas for future research. It is not 
intended to be exhaustive by any means. In order to keep tiiis section reasonably brief, procedures which 
differ from the present paper's focus on distance-based measures are not discussed. Examples of other 
approaches not considered here include non-distance similarity coefficients (e.g., q-correlations) and 
algorithms which combine specific variable weighting or multidimensional scaling models with specific 
clustering algorithms. 

While restricting the present investigation to bivariate data somewhat limits its generalizability, it 
was felt that this provides minimal test of the weighting procedures; any procedure which does not function 
well with bivariate data seems unlikely to be useful for higher dimensional cases. Having established here 
that variable weighting methods can provide substantial benefit in cluster recovery, further work is needed 
to determine the extent to which the results obtained here generalize to higher numbers of subgroups and 
higher dimensional clustering problems. 

Certain aspects of the methods used, such as how to choose the number of distances m to use in 
computing the ACE estimate of S, need to be addressed. Currently, there is little to guide the applied 
researcher. J. Kettenring (personal communication, July 2, 1993) indicates that some work along these 
lines is underway at Bellcore. Obviously, more work would be useful. A related ' sue is the relationship 
of the original formulation of the method, used here, to the somewhat modified version which is 
implemented in SAS’s PROC ACECLUS (1988). 

The relatioaship between th ' methods used here and procedures a’mcd at variable standardization 
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(Milligan & Cooper, 1988; Barton, 1993) should be examined. Variable standardization methods seek to 
deal with problems associated with combining variables measured on very different scales (such is SAT 
and GPA) into a single similarity coefficient. Hence, variable standardization procedures tend to treat 
variables sequentially, as opposed to the multivariate focus of the procedures examined in the present 
study. 

Of the clustering methods examined in this study, the flexible average clustering method (Belbin, 
Faith, & Milligan, 1992) gave tiie best recovery of the methods in this study. The optimal value of P was 
found to be -. 15 to -.20, which is slightly lower than the value of P = -. 10 suggi>*ted by Belbin et al. 
(1992). A companion paper (Donoghue, 1995) presents further results comparing the flexible average and 
beta-flexible methods. Of the remaining methods. Ward’s method and beta-flexible linkage (P = -.50) 
yielded die best recovery, followed closely by SAS's EML algorithm. 

The relationship of the weighting methods identified here to the closely allied problem of variable 
selection (Milligan, 1989b; Donohue, 1994a) remains to be addressed. Both of these papers found that 
fee ultraraetric weights were useful in ameliorating the degradation of cluster recovery caused by including 
irrelevant variables in fee cluster analysis. Donoghue (1994a) also compared two additional variable 
selection procedures, and in their original paper. Art et al. (1982) presented preliminary evidence that fee 
ACE weighting method may be useful with the variable selection problem. More work is needed to 
understand fee interrelationship of the various variable weighting techniques proposed to solve different 
problems in clustering. 



Conclusion 

Tliis study was meant to constitute a first step in imderstanding fee utility of various variable 
weighting algorithms. The results of Donoghue (1994b) clearly indicate feat cluster analyses based upon 
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tile Euclidean distance can yield suboptimal results in the presence of within-group correlation, but could 
not offer clear suggestions for alternatives. 

Variable weighting metiiods were found to have a large effect on cluster recovery. Of the 
metiiods which do not require a priori knowledge of the subgroup structure, the ACE method of Art, 
Gnanadesikan, and Kettenring (1982) yielded the best results. When used with the better clustering 
algorithms, this method provided a net improvement in 17% (EML) to 24% (beta-flexible, P = -.50) of 
the datasets examined. When used with the same clustering methods, the next best weighting method, 
ultrametric wei^ts, yielded improved recovery 16-21% of the time. However, ultrametric weighting was 
found to be more sensitive than ACE to within-subgroup correlation. Therefore, ACE is tiie metiiod 
which is preferred overall. There is still plenty of room for improvement, however. Comparisons of 
ACE witii Dm based on the pooled within-group covariance matrix indicated that knowing the correct 
covariance matrix would yield improved recovery (over ACE) approximately 10% of the time. 

Finally, it should be noted that the procedures examined and the results obtained here only apply to 
situations in which a mixture model formulation of the clustering problem makes sense. In some 
educational applications, such as clustering test items based on multidimensional IRT parameters (Miller & 
Hirsch, 1992), it is far from obvious whether the mixture model formulation of the clustering problem is 
valid. The application of the present work to such situations is problematic. 

This study is intended to be a first step in understanding variable weighting methods. It attempts to 
provide some information to those educational investigators faced with doing empirical cluster analyses, 
allowing them to make a better informed choice of variable weighting methods and clustering algorithms. 
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Table 1 

Descriptive Statistics for Observed Dm 



1 Dm_ 


Mean 


Std. 


Min. 


Q, 


Med. 


0, 


Max. 


2 


2.086 


0.454 


0.372 


1.835 


2.051 


2.300 


4.611 


4 


4.106 


0.558 


2.127 


3.775 


4.063 


4.357 


7.185 


6 

' 


6.143 


0.730 


3 . 5:1 


5.691 


6.067 


6.500 


10.690 
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Table 2 

Main Effects and Salient Interactions ^ .03) for ANOVA of HA-Rand Index 



EFFECT 


DF 


SS 


T1\h 


D 


2. 


23826.77 


.641 


P 


1. 


59.61 


.004 


C 


4. 


648.85 


.046 


R1 


2. 


425.51 


.031 


R2 


2. 


97.03 


.007 


M 


10. 


4166.77 


.238 


W 


5. 


1452.93 


.098 


D*C 


10. 


540.93 


.039 


D*M 


20. 


702.38 


.050 


P*M 


10. 


723.06 


.051 


C*M 


40. 


436.27 


.032 


C*R1 


8. 


337.65 


.025 


D*W 


10. 


332.11 


.024 


P*W 


5. 


671.72 


.048 


C*W 


20. 


834.55 


.059 


Rinv 


10. 


568.05 


.041 


I R2*W 


10. 


83.78 


.006 


_ . 

M*W 


50. 


631.73 


.045 


D*P*C 


8. 


414.63 


.030 


D*M*W 


100. 


328.51 


.024 


p»M*W 


50. 


312.40 


.023 


Error 


338580. 


13353.61 


— 



Abbreviations: D - Mahalanobis distance between subgroup centroids, P - probability of subgroup 
membership, C - covariance matrix condition, R1 - correlation in larger group 
(when Pj pj), R2 - correlation in smaller group (when Pi ^ Pj), M -- method of 
clustering, W - weighting method. 
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Table 3 

HA-Rand Index Means and (Std. Dev.) for Selected Effects 
Djj Prob. 



2 


4 


6 


Pi = 0.5 


Pi =0.9 


.241 (.249) 


.637 (.362) 


.867 (.283) 


.595 (.3%) 


.569 (.399) 



COV 



1:1 


1:9 


9:1 


HO 


HI 


.594 (.405) 


.636 (.354) 


.584 (.441) 


.505 (.410) 


.590 (.395) 



R1 R2 



-.7 


0.0 


0.7 


-.7 


0.0 


0.7 


.533 (.411) 


.607 (.389) 


.606 (.387) 


.559 (.408) 


.591 (.394) 


.5% (.389) 



Weight 



Euclid 


Dm-T 


Dm-W 


PC-R 


ACE 


Ultrametric 


.512 (.418) 


.492 (.400) 


.677 (.352) 


.583 (.402) 


.627 (.381) 


.599 (.397) 



Method 


Mean Recovery 


Single Link (Si) 


.318 (.414) 


Complete Link (Cm) 


.456 (.390) 


Ward’s Method (Wa) 


.623 (.385) 


EML 


.609 (.389) 


Average Linkage (Avg.) 


.515 (.422) 


Flexible Average, P = -.10 (AvlO) 


.663 (.365) 


Flexible Average, P = -.15 (Avl5) 


.682 (.353) 


Flexible Average, P = -.20 (Av20) 


.682 (.353) 


Flexible Average, P = -.25 (Av25) 


.673 (.359) 


1 Beta Flexible, p = -.25 (Fx25) 


.568 (.388) 


1 Beta Flexible, P = -.50 (Fx50) 


.609 (.381) 
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Table 4 

Ranks of Weighting Methods Based on Ordinal Comparisons 
(by Clustering Method) 



Weighting Method 


Si 


Cm 


Wa 


EML 


Avg. 


AvlO 


Avl5 


Av20 


Av25 


Fx25 


Fx50 


Dm-W 


1.5'^ 


1 


1 


1 


1 


1 


1 


1 


1 


1 


1 


ACE 


1.5" 


2 


2 


2 


2 


2.5* 


2 


2 


2 


2 


2 


Ultrametric 


3 


3 


3 


3 


3 


2.5* 


3 


3 


3 


3 


3 


PC-R 


4 


4 


4 


4 


4 


4 


4 


4.5" 


4.5*^ 


4 


4 


Dm-T 


6 


6 


5 


6 


6 


5 


5 


4.5" 


4.5^^ 


6 


5.5" 


Euclid 


5 


5 


6 


5 


5 


6 


6 


6 


6 


5 


5.5" 



Entries with common superscripts do not significantly differ from one another. 
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Table 5 

Values for Selected Comparisons of Weighting Methods 
(by Clustering Method) 





Euclid 


ACE 


Dm- 


W 




Dm-T 


PC-R 


Ultra. 


ACE 


Dm-W 


Dm-T 


PC-R 


Ultra. 


ACE 


Ultra. 


Si 


.085 


-.031 


-.053 


-.081 


-.069 


.169 


.050 


.026 


-.007* 


.017 


Cm 


.275 


-.121 


-.182 


-.234 


-.417 


.486 


.123 


.053 


.184 


.251 


Wa 


-.070 


-.133 


-.168 


-.230 


-.331 


.166 


.104 


.067 


.112 


.176 


EML 


.165 


-.147 


-.160 


-.168 


■211 


.342 


.046 


.025 


.109 


.134 


Avg. 


.213 


-.097 


-.120 


-.132 


-.224 


.350 


.043 


.010* 


.096 


.109 


AvlO 


-.034 


-.148 


-.162 


-.160 


-.280 


.140 


.025 


.016* 


.128 


.135 


Avl5 


-.113 


-.160 


-.163 


-.184 


-.296 


.086 


.048 


.033 


.130 


.160 


Av20 


-.127 


-.155 


-.176 


-.210 


-.306 


.091 


.069 


.052 


.116 


.165 


Av25 


-.121 


-.139 


-.171 


-.222 


-.303 


.104 


.086 


.061 


.103 


.155 


Fx25 


.114 


-.154 


-.211 


-.239 


-.393 


.348 


.114 


.066 


.161 


.214 


Fx50 


-.010* 


-.144 


-.174 


-.244 


-.366 


.246 


.119 


.072 


.118 


.199 

. 



“ Does not significantly differ from 0.0, p — .05. 
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Table 6 

Ranks of Clustering Algorithms Based on Ordinal Comparisons 
(by Weighting Method) 





Euclid 


Dm-T 


Dm-W 


PC-R 


ACE 


Ultrametric 


Av20 


2"* 


1.5^ 


1* 


2 


1" 


1.5" 


Avl5 


1" 


1.5^ 


2 '" 


1 


3" 


1.5" 


AvlO 


36c 


4 


4 


3 . 5 ' 


5^ 


4^ 


Av25 


4^ 


3 


3' 


3 . 5 ' 


2”° 


3^ 


Wa 




5 


5.5^' 


6.5"‘ 


5^ 


6' 


Fx50 


T 


6 


5.5^' 


6.5'” 


5^ 


6‘ 


EML 


5“^ 


7 


7.5* 


5 


7 


6‘ 


Fx25 


8 


8 


7.5* 


8 


8 


8 


Avg. 


9 


9.5* 


10 


9 


9.5*^ 


9 


Cm 


10 


9.5* 


9 


10 


9.5*^ 


10 


Si 


11 


11 


11 


11 


1! 


11 



■4 ( 



ahcdcfghijkimiiopqrsf (.(jmmon superscripts do not significantly differ from one another. 



Table? 

Values for Selected Comparisons of Clustering Methods 
(by Weighting Method) 



Comparison with Flexible Average, P = -. 1 5 








Si 


Cm 


Wa 


EML 


Avg. 


AvlO 


Avl5 


Av20 


Av25 


Fx25 


Fx50 1 


Euclid 


.440 


.394 


.142 


.131 


.247 


.030 


— 


.009* 


.025 


.221 


.137 


Dm-T 


.723 


.689 


.159 


.385 


.566 


.107 


— 


-.006* 


.017* 


.442 


.259 


Dm-W 


.613 


.295 


.088 


.118 


.274 


.034 


— 


-.00;.* 


.011* 


.131 


.079 


PC-R 


.536 


.411 


.136 


.123 


.273 


.035 


— 


.016 


.034 


.216 


.150 


ACE 


.591 


.326 


.076 


.116 


.296 


.055 


— 


-.018 


-.010* 


.142 


.069 


UltTfcinetric 


.521 


.359 


.114 


.114 


.262 


.021 




-.003* 


.010* 


.180 


.116 


Comparison with Ward’s Method 


Euclid 


.319 


.270 


— - 


.005* 


.122 


-.109 


-.142 


-.126 


-.110 


.086 


-.005* 


Dm-T 


.543 


.546 




.251 


.431 


-.044 


-.159 


-.173 


-.148 


.285 


.077 


Dm-W 


.545 


.212 


.... 


.026 


.200 


-.054 


-.088 


-.087 


-.065 


.028 


-.012* 


PC-R 


.386 


.273 


— - 


-.008* 


.143 


-.102 


-.136 


-.124 


-.104 


.071 


-.010* 


ACE 


.541 


.274 


— 


.058 


.246 


-.011* 


-.076 


-.096 


-.095 


.076 


-.015* 


Ultrametric 


.416 


.259 


— 


.008* 


.158 


-.087 


-.114 


-.116 


-.110 


.059 


-.005* 



* Does not significantly differ from 0.0, p = .05. 
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A2 




Figure Captions 



Figiifft 1 ■ Plot of mean cluster recovery (HA-Rand) for two-way interaction of correlation in the 
larger groi^j (Rl) with variable weighting metiiod (Weight). 

Fi gure 2. Plot of mean cluster recovery (HA-Rand) for two-way interaction of covariance 
condition (COV) with variable weighting mefliod (Weight). 

Figure "i. Plot of mean cluster recovery (HA-Rand) for two-way interaction of subgroup size 
(probability of subgroup 1 membership— Pi) with variable wei^ting method (Weight). 

Fi gure 4 . Plot of mean cluster recovery (HA-Rand) for two-way interaction of variable weighting 
method (Weight) with clustering method (Method). 

Eigurfi_5. Plot of mean cluster recovery (HA-Rand) for two-way interaction of Mahalanobis 
distance between the centroids of the subgroups (Distance) with clustering metiiod (Method). 

Fi gure 6 . Plot of mean cluster recovery (HA-Rand) for two-way interaction of subgroup size 
(probability of subgroup 1 membership— p,) with clustering method (Method). 

Figure 1 . Plot of mean cluster recovery (HA-Rand) for two-way interaction of covariance 
condition (COV) witii clustering metiiod (Method). 

Fi gure R Plot of mean cluster recovery (HA-Rand) for three-way interaction of Mahalanobis 
distance between the centroids of the subgroups (Distance) with covariance condition (COV) with 
subgroup size (probability of subgroup 1 membership-pi). 

a) Distance by COV interaction for equal-sized groups (pi=.5). 

b) Distance by COV interaction for unequal-sized groups (p, = .9). 
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R1 by Weight Interaction 




Weighting Method 




COV by Weight Interaction 




Weighting Method 




Prob. by Weight Interaction 




Recove 



Weight by Method Interaction 




:k -f 



Dist. by Method Interaction 




Clustering Method 



Mahalanobis Distance 
-- Dm " 2 — Dm “• 4 



Dm = 6 






O 

ERIC 



53 



Prob. by Method Interaction 





COV by Method Interaction 




Clustering Method 



Dist. by COV Interaction 



a) p1 = 0.5 




Mahalanobis Distance 

Dm = 2 Dm = 4 + Dm = 6 



Dist. by COV Interaction 



b) p1 = 0.9 




Mahalanobis Distance 

Dm “2 Dm = 4 Dm -6 



O 
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Captions for Additional Figures 



. igiire Al. Plot of mean cluster recovery (HA-Rand) for tiiree-way interaction of Mahalanobis 
distance between the centroids of tite subgroi^ps (Distance) with variable weighting method 
(Weigjit) wifli clustering method (Method). 

a) Wei^t by Mefliod interaction for D^, = 2.0. 

b) Wei^t by Method interaction for D^, = 4.0. 

c) Weight by Method interaction for D^, = 6.0. 

Figure A2. Plot of mean cluster recovery (HA-Rand) for three-way interaction of subgroup size 
(probability of subgroup 1 membership— pi) widi variable weighting method (Weight) witih 
clustering method (Mefood). 

a) Weight by Method interaction for equal-sized groups (pi = .5). 

b) Wei^t by Method interaction for imequal-sized groups (Pi= .9). 




Weight by Method Interaction 



a) Dm = 2.0 




Clustering Method 



Weight by Method Interaction 




h'l 



o 

ERIC 



Weight by Method Interaction 



c) Dm = 6.0 
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Weight by Method Interaction 



a) p1 = 0.5 




Si Cm Wa EML Avg. Av10 Av15 Av20 Av25 Fx25 Fx50 

Clustering Method 



Weight by Method Interaction 



b) p1 = 0.9 




o 



